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Intellectual Property Rights 



IPRs essential or potentially essential to the present document may have been declared to ETSI. The information 
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found 
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in 
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web 
server ( http://webapp.etsi.org/IPR/home.asp) . 

Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee 
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web 
server) which are, or may be, or may become, essential to the present document. 



Foreword 

This Technical Specification (TS) has been produced by ETSI User Group (USER). 

Information and Communication Technology (ICT) standardisation is part of the general standardisation activities, and 
contributes to policy objectives to improve the competitiveness of European industry, as specified in the Lisbon 
strategy. The legal basis for European standardisation and standardisation policy, including the ICT domain, is 
Directive 98/34/EC [i.2]. One of its main elements is the formal recognition of three European Standards 
Organisations (ESO), CEN, CENELEC and ETSI, active in various degrees in the ICT domain. Standards produced by 
the three ESO and resulting from an open consensus building process are by nature voluntary and non binding technical 
documents. 

The standardization work described in the present document was funded by the European Commission, Enterprise and 
Industry Directorate-General, as part of the 2009 ICT Standardisation Work Programme and executed by ETSI. 

The present document has been produced within the ETSI Special Committee USER GROUP (USER) by the Specialist 
Task Force (STF) 374. 



Introduction 

In order to provide evidence that the Quality of Service (QoS) assessment is conducted according to TS 102 852 [1], it 
will be useful for a service provider to have its QoS assessment process audited by an independent party. The purpose 
of the present document is to define when, how and by whom a conformity assessment audit of the QoS assessment 
shall be conducted. Such a conformity assessment is expected to contribute to an increased confidence by service 
providers, customers, regulators and other stakeholders in QoS assessment results. 

The present document provides the requirements needed to ensure that QoS information has been assessed according to 
the best practices and to TS 102 852 [1]. It can be used by any assessment body to assess the conformity of the 
assessment process of the Quality of Service (QoS) of Services Providers (SP). 

NOTE: Conformity assessment against the present document may be issued by a certification body accredited 
ISO/IEC Guide 65 [2] by a European Cooperation for Accreditation (EA) member 
( www.european-accreditation.org) . EA members are listed in Annex A. In that case, the declaration of 
conformity is called a "certificate of conformity " . 

Although the present document is not a multipart deliverable, it is closely linked to: 

• The EG 202 843 [i.l] that defines parameters and basic information which are universally applicable. One aim 
here is to keep the parameter definitions stable and complete for any kind of application. 

• The TS 102 852 [1] that defines the test methodologies, how to apply the parameters including all necessary 
boundary conditions and preconditions with the aim to ensure comparability of results and to guarantee the 
objectivity of the results. 



ETSI 



ETSI TS 102 844 V1.1.1 (2010-10) 



Scope 



The present document details the specific requirements an independent body should comply with in addition to the 
generic standards ISO/IEC Guide 65 [2] to provide conformity assessment of the QoS assessment process of 
Information and Communication Technology (ICT) services to EG 202 843 [i.l] and other relevant standards listed in 
the present document. 

NOTE 1: According to TS 102 852 [1], QoS of the customer Relationship Stages other than utilization is assessed. 

NOTE 2: The present document may be used by any stakeholder concerned by QoS Assessment in order to be 
confident with the results of QoS campaigns and with the way they are carried out. 



References 



References are either specific (identified by date of publication and/or edition number or version number) or 
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the 
reference document (including any amendments) applies. 

Referenced documents which are not found to be publicly available in the expected location might be found at 
http://docbox.etsi.org/Reference . 

NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee 
their long term validity. 

2.1 Normative references 

The following referenced documents are necessary for the application of the present document. 

[1] ETSI TS 102 852: "User Group; Quality of ICT Services; Assessment process of the QoS 

parameters of the customer relationship stages". 

[2] ISO/IEC Guide 65 (1996): "General requirements for bodies operating product certification 

systems". 

NOTE: ISO/IEC Guide 65 is expected to be replaced by ISO/IEC 17065 next year. 

[3] ISO 19011: "Guidelines for quality and/or environmental management systems auditing". 

[4] CENELEC EN 45020 (1998): "Standardisation and related activities - General vocabulary". 

[5] CENELEC EN 4501 1 : "General requirements for bodies operating product certification systems". 

NOTE: CENELEC EN 4501 1 is expected to be replaced by ISO/IEC 17065 next year. 

2.2 Informative references 

The following referenced documents are not necessary for the application of the present document but they assist the 
user with regard to a particular subject area. 

[i.l] ETSI EG 202 843: "User Group; Quality of ICT Services; Definitions and Methods for Assessing 

the QoS parameters of the Customer Relationship Stages other than utilization". 

[i.2] Directive 98/34/EC of the European Parliament and of the Council of 22 June 1998 laying down a 

procedure for the provision of information in the field of technical standards and regulations and of 
rules on Information Society services. 
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3 Definitions and abbreviations 

3.1 Definitions 

For the purposes of the present document, the following terms and definitions apply: 

accreditation body: organization that has the power to accredit a Conformity Assessment Body 

audit team: group of auditors appointed to provide conformity assessment of the QoS assessment activities with 
respect to a technical specification 

auditor: person who is competent to carry out audits 

conformity assessment: activity to determine, directly or indirectly, that a process, product, or service meets relevant 
standards and fulfils relevant requirements 

NOTE: Typical examples of conformity assessment activities are sampling, testing and inspection; evaluation, 
verification and assurance of conformity (supplier's declaration, certification); registration and approval 
as well as their combinations EN 45020 [4], clause 12.2. 

Conformity Assessment Body (CAB): organization that has skills and methods to conduct conformity assessment 
audit of the QoS assessment process of another party 

deviation: requirement that is not or partially fulfilled 

NOTE: A deviation may be major or minor. 

3.2 Abbreviations 

For the purposes of the present document, the following abbreviations apply: 

CAB Conformity Assessment Body 

EA European Cooperation for Accreditation 

ESO European Standards Organisation 

ICT Information and Communication Technology 

QoS Quality of Service 

QoSAP Quality of Service Assessment Party 

SP Service Provider 



4 Approach for Conformity Assessment 

4.1 Scope of Conformity Assessment 

The Conformity Assessment process described in the present document is concerned with the compliance of the QoS 
assessment process managed by a service provider or by an independent party with requirements described in 
TS 102 852 [1]. The intent of the Conformity Assessment described in the present document is not to state the level of 
QoS reached by the service provider but the conformity of the QoS assessment process carried out by the service 
provider or by the QoSAP with TS 102 852 [1]. 
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4.2 Types of Conformity Assessment 

It is expected that some service providers may outsource the QoS assessment to an external third party and that other 
service providers may handle it internally. 

In order to address these two schemes, two types of conformity assessment have been defined: 

a) External QoSAP. When an external QoSAP carries out the QoS Assessment Process for several service 
providers, the conformity assessment of the QoSAP can be carried on the basis of generic methods used by the 
QoSAP and examples of real cases of QoS assessment of different service providers. This is called a type A 
conformity assessment. 

b) Internal QoSAP. When a service provider executes with its own means the QoS assessment of its services, 
the conformity assessment is carried out on the basis of the methods used to assess the QoS. This is called a 
type B conformity assessment. 

With this flexible scheme, when a service provider wants to make stakeholders confident that the QoS assessment is 
carried out according to TS 102 852 [1], he will have the choice between: 

• subcontracting the assessment process to an external QoSAP which has an up-to-date generic type A 
conformity assessment awarded by a CAB (Conformity Assessment Body); or 

• executing by its own means the QoS assessment process and hiring a CAB to receive and update a specific 
type B conformity assessment. 

NOTE: Conformity to the present document is applicable only if the QoSAP carries out the totality of the 
assessment process described in TS 102 852 [1]. 

Figure 1 illustrates the different roles and actions of entities involved in the conformity assessment Process. 




Figure 1 : Role of Entities and Applicable Standards in Conformity Assessment of 
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Figure 2: The QoSAP type A (External QoSAP) 
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Figure 3: The QoSAP type B (Internal QoSAP) 



Scope of conformity assessment 



The QoSAP shall provide the audit team of the CAB with real QoS Assessment campaigns carried out by the QoSAP. 

The auditing may be carried out in part/s on different cases to meet the complete requirements. No case/s need to be 
audited in full. 



6 Audit team for conformity assessment 

The audit team may be composed by a sole or several auditors. Each of them shall be qualified by the party providing 
assessment of the QoS. In case of several auditors, one shall be designed as the audit team leader. Auditor shall have: 

• Secondary education or 7 years of professional experience. 

• Understanding of statistics. 

• Knowledge in ICT. 

• Basic knowledge of Mechanism for ICT QoS assessment. 

The audit team leader shall have experience in audit as lead auditor as described in ISO 1901 1 [3]. 

Auditors shall maintain their qualification on an annual basis through active participation in assessment activities in 
QoS Assessment process and updating knowledge in the relevant standards. 



7.1 



Audit activities 



Overview 



The activities of the audit of the QoS assessment are the following 

• Initiating the audit. 

• Conducting document review. 

• Preparing for the on-site audit activities. 

• Conducting on-site audit activities. 
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• Preparing the audit report. 

• Approving and distributing the audit report. 

• Completing the audit. 

• Conducting audit follow-up. 

Additional information about audit activities can be found in ISO 1901 1 [3]. 
Decision related to conformity is declared by the CAB. 

7.2 Audit Methods 

Three levels are defined to assess the compliance of the QoS Assessment process: 

• Inquiry: interview of the personnel in charge of the ICT QoS assessment. 

• Examination: review of documentation, visual examination. 

• Tests: test or computation of results. 

The means of assessment is chosen by the auditor. It may be different for each requirement. 

7.3 Assessment of QoSAP Status 

The QoSAP that conducts the QoS Assessment process shall employ sufficient and competent personnel to perform the 
checking process as described in TS 102 852 [1]. This staff shall have the following minimum skills, knowledge and 
competences: 

Specialist knowledge in statistics. 

Specialist knowledge in ICT. 

Specialist knowledge in customer relationship. 

Specialist knowledge of design and analysis of surveys and assessment methods for customers viewpoint. 

Specialist knowledge in QoS assessment methods and tools. 

NOTE: "Specialist" means that the personnel should have a specialized education and/or experience. 



7.4 Independency 



The QoSAP shall demonstrate that conflicts of interest do not exist with the departments in charge of managing the QoS 
within the service providers. 



8 Assessment of the campaign process 

The QoSAP shall have a documented process to manage the test campaign. The process shall include at least the stages 
detailed in TS 102 852 [1]: 

a) Define campaign. 

b) Build questionnaire. 

c) Prepare campaign operation. 

d) Execution of campaign. 

e) Data filtering. 
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f) Aggregation of results. 

g) Result Analysis. 

h) Report, representation of results. 



9 Assessment of the campaign definition and operation 

For each campaign, the QoSAP shall document the campaign according to TS 102 852 [1]. 
The following information shall be provided by the QoSAP: 

a) Statistical considerations used to design the test campaigns and the sampling in accordance with 
TS 102 852 [1]; the present document shall describe elements such as : 

criteria in the choice of data sources, expert panel and/or consumers; 

criteria used to define the minimum number of persons in the panels and their profiles; 

way to define the detailed tasks for expert panel. 

b) Rationale for relevance and completeness of expertise of the panel of customers and/or experts: 

method used to design the questionnaires in accordance with TS 102 852 [1]; 

method used to prepare the test campaign and the sampling in accordance with TS 102 852 [1]. 

c) Capability of the means used to execute the campaign. 

d) Methods used to collect information from panel. 

e) Methods used to filter data. 

f) Methods used to aggregate the results. 

g) Methods used to analyse the results. 

h) Frame work/s for representation of the results. 

1 Assessment by the audit team 

The audit team shall verify that documented methods provided by the QoSAP are correctly used (on real cases) and 
maintained in a way that ensure that results are objective, comparable and reproducible. 

To be declared compliant, the QoSAP shall provide the audit team at least one real cases of QoS assessment. If the 
QoSAP has no real cases to provide, it can provide a case study. The QoSAP will then be "provisional compliant". It 
will be fully compliant as soon as it has a real case to provide to the CAB. 

1 1 QoSAP Resources and testing equipment 

The QoSAP shall provide the list of resources, tools and configuration equipment available for QoS assessment. 



12 Audit report 



The audit report re-state conclusions for each recommendation from EG 202 843 [i.l] and requirement from 
TS 102 852 [1]. It identifies for each of them: 

• activities performed; 
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• Service Provider name (s) (in case of external QoSAP); 

• time period; 

• locations; 

• rationale of efficiency and findings. 

It also states a general conclusion of the assessment and recommendations for certification/compliance. 

12.1 Audit findings and non conformities 

Any deviation from TS 102 852 [1] or from the present document requirements related to the QoS assessment process 
shall be identified as a non conformity. The CAB is free to define different grades of deviations. 

Certification/conformity shall not be granted until all requirements are met. Non conformities, which raise any doubt on 
the capability of QoSAP to carry out QoS assessment in a correct manner must be corrected and the correction verified 
by the CAB (by site visit or other appropriate forms of verification) before a certification/ conformity is granted. 



1 3 Monitoring the conformity 



TS 102 852 [1] certificate of conformity is valid for 5 years. Periodic surveillance of QoSAP activities shall be 
implemented through periodic audits in order to verify that the QoSAP continue to comply with TS 102 852 [1]. For the 
first period, a surveillance audit must be carried out within 12 months after the initial assessment. This frequency may 
be maintained or reduced if the number of non conformities found during the latest audit raise doubt on the QoS AP's 
ability to maintain its conformity. The frequency can be extended to 1 8 months if no non conformity has been identified 
at the latest audit. 

The QoSAP shall inform the CAB of every change that occurred in its organisation. 



14 Audit duration 

The duration of the audit (initial, surveillance and renewal) shall be determined according to table 1 . 

Table 1 : Audit duration 



Total days 


QoSAP Number of employees involved in QoS 
assessment 


Preparation and report : 0,5 
On site audit : 1 


1 to 5 


Preparation and report : 0,5 
On site audit : 2 


6 to 10 


Preparation and report : 1 
On site audit : 2 


1 1 to 25 


Preparation and report : 1 
On site audit : 3 


26 to 45 


Preparation and report : 1 ,5 
On site audit : 3 


46 to 65 


Preparation and report : 1 ,5 
On site audit : 4 


IVIore tiian 70 
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Annex A (informative): 

List of European accreditation bodies members of EA 

(European Cooperation for Accreditation) 

This list provided for information is an excerpt of http://www.european-accreditation.org/nl/doc/EA- 1 -05 .pdf dated on 
23 December 2009. 

For an up-to-date list check http://www.european-accreditation.org/content/ea/members.htm 



ACCREDIA : 

BAS: 

BELAC : 

BMWFJ: 

CAI : 

CQFRAC : 

COPA: 

CYS-CYSAB: 

DAkkS: 

DANAK : 

EAK: 

ENAC : 

ESYD : 

FINAS : 

HAA: 

lARM: 

INAB : 

IPAC : 

ISAC: 

LA: 

LATAK: 

NA : 

NAB: 

NAT: 

PL AS : 

PCA : 

RENAR : 

RvA: 

SA: 
SAS : 

SNAS : 
SWEDAC : 
TURKAK : 
UKAS: 



Italian Accreditation System (ITALY) 

Executive Agency "Bulgarian Accreditation Service" (BULGARIA) 

Belgian Accreditation Structure (BELGIUM) 

Bundesministerium fiir Wirtschaft Familie und Jungend, (AUSTRIA) 

Czech Accreditation Institute, (Cesky Institut pro Akreditaci, o.p.s. - CZECH REPUBLIC) 

Comite Francais d' Accreditation (FRANCE) 

Consorzio Pubblico per 1' Accreditamento (ITALY) 

Cyprus Organization for the Promotion of Quality (CYPRUS) 

DeutscheAkkreditierungsstelle GmbH (GERMANY) 

Danish Accreditation (DENMARK) 

Estonian Accreditation Centre (ESTONIA) 

Entidad Nacional de Acreditacion (SPAIN) 

Hellenic Accreditation System S.A. (GREECE) 

Finnish Accreditation Service (FINLAND) 

Croatian Accreditation Agency (REPUBLIC OF CROATIA) 

The Accreditation Institute of the former Yugoslav Republic of Macedonia (REPUBLIC OF 

MACEDONIA) 

Irish National Accreditation Board (IRELAND) 

Portuguese Institute for Accreditation (Portugal) 

Icelandic Board for Technical Accreditation (ICELAND) 

Lithuanian National Accreditation Bureau (LITHUANIA) 

Latvian National Accreditation Bureau (LATVIA) 

Norwegian Accreditation (NORWAY) 

National Accreditation Board -Malta (MALTA) 

Hungarian Accreditation Board (HUNGARY) 

Office Luxembourgeois d' Accreditation et de Surveillance (LUXEMBURG) 

Polish Centre for Accreditation (POLAND) 

Romanian Accreditation Association (Asociatia de Acreditare din Romania - ROMANIA) 

Dutch Accreditation Council (Raad Voor Accreditatie - NETHERLANDS) 

Slovenska Akreditacija (SLOVENIA) 

State Secretariat for Economic Affairs (SECO), Swiss Accreditation Service (SWITZERLAND) 

Slovak National Accreditation Service (SLOVAKIA) 

Swedish Board for Accreditation and Conformity Assessment (SWEDEN) 

Turkish Accreditation Agency (TURKEY) 

United Kingdom Accreditation Service (UNITED KINGDOM) 
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